Real-time AR content management and intelligent data analysis system

ABSTRACT

Described is a real-time content management and data analytics system for AR-based platforms. The data management system described herein manages AR content and user interactions with the AR content. Additionally, a new multiple-stage information augmentation design based on real-time data analysis and live AR interaction is described. In this design, AR content design is very flexible and may be organized into one or more stages containing pre-defined content, on-line searched content, user generated content, other user generated content, real-time user interactively generated content, or some combination. The flexibility content structure brings allows for a high customizable AR experience for each user to maximize user relevance and stickiness. An example application of the system described herein is for interactive advertisements.

CROSS REFERENCE TO RELATED APPLICATIONS

The application claims priority to U.S. Provisional Patent ApplicationNo. 62/502,646 filed on May 6, 2017 entitled “Real-time AR ContentManagement and Intelligent Data Analysis System”.

FIELD OF INVENTION

The present invention is related to the field of augmented reality. Moreparticularly, it concerns systems, devices, and methods for managing ARcontent and conducting data analytics on AR content and user activitywithin an AR platform. Additionally, the invention concerns datastructures for AR content. Specifically the invention provides a newdata structure that renders an AR environment from one or more stages ofAR content.

BACKGROUND

Mixed reality (MR) is the merging of real and virtual worlds to producenew environments and visualizations where physical and digital objectsco-exist and interact in real time. MR combines augmented reality (AR)techniques with virtual reality (VR) method to deliver an immersiveexperience. The goal of MR is to provide a rich, real-time interactiveexperience for the user that allows them to feel as if they were in aplace that is different from their actual physical location.Applications of MR are widespread and diverse, but each is based on theunderlying concept of receiving real world sensory input, for example,sound, video, graphics, haptics, or location data and adding furtherdigital insights to that information.

Recent technological advancements and social changes have broughtMR/AR/VR technology into the mainstream. Powerful mobile computingdevices, such as smart phones, tablet computers, laptops, and navigationdevices provide the necessary hardware to support more advancedapplications for processing real word sensory input. Additionally,widespread use and adoption of specialized computer applications andartificial intelligence has generated significant consumer andprofessional interest in using AR platforms to provide a more realisticsimulation of a real world experience. This abundance of computingdevices and growing interest from business and the public has spawned aperfect storm for the development and adoption of MR/AR/VR platforms.

Holographic MR is an emerging type of MR experience. Holographic MRenvironments display fully three-dimensional objects within an ARenvironment. Until recently, these complex systems where impossible toprovide at scale because of computational resource limitations. Acontinued rise in performance of computer memory and graphicalprocessing speed in accordance with Moore's Law, however, is bringingscalable holographic MR systems within the realm of possibility. Despitemuch excitement about- and development on-holographic MR systems most ofthe infrastructure necessary to realize scalable MR has not beencompleted. Accordingly, there exists a well-established need for contentmanagement systems that efficiently provide AR content to a holographicMR environment.

Despite the technological potential of advanced MR/AR systems most ARenvironments render digital objects within a real environment. Thesesystems allow users to picture impossible, expensive, or hard to findvirtual objects in an a familiar environment, for example, the home,school, walk to work, or other common scenes from everyday life. Inother words most AR systems bring the virtual world into the immediatelyaccessible real world environment. Currently, there is no possibilityfor actual real life objects, including people, pets, and personalpossessions to travel to- and interact with- a virtual world.Accordingly, there exists a well-established need for an contentmanagement system that manages MR/AR/VR content in a way that provideslive user interaction with a virtual environment. Similarly, thereexists a well-established need for a content structure that can beeasily managed within a content management system to provide real lifeimages and live user interactions within a virtual environment. The userinteractions should be fully dynamic and the content structure should beflexible enough to allow different user interactions with the sameMR/AR/VR environment. Additionally, there exists a well-established needfor a content management system that incorporates a real time dataanalytics module that can modify or update and AR environment accordingto user preferences or analytics data. Moreover, there exists awell-established need for an AR content structure that can beefficiently managed to incorporate real life images and live userinteractions as well as real time data analytics information. Thecontent structure should allow some parts of the AR content to bemodified without effecting the remaining portions. The transitionsbetween items of AR content or portions of the same AR content shouldalso be independently modifiable.

SUMMARY

The present invention is related to systems, devices, and methods foraugmented reality (AR) environments. More particularly, the inventionrelates to content management systems for managing content within anMR/AR/VR platform. Content managed by the content management systemincludes AR content, for example AR source content, AR user content, ARcombined content as well as non AR content, for example, images, video,audio, and analytics information. The content management system managesmultiple kinds of AR content during the AR process to provide relevantand real-time responsive and interactive AR content to the user.Additionally, the AR content management system incorporates data fromthird party services, for example, business intelligence providers andsocial medial networks, to manage advertisement bidding and immersion aswell as social media sharing and responses in real time.

Artificial intelligence and computer vision are deployed in AR contentmanagement system described herein as well as data analytics and liveuser interactions. In one example, the content management and dataanalytics system also enables an offline AR system to immerse AR contentinto online social media networks and other real-time Internetapplications. Another embodiment of the invention enables immersion ofsocial media information and other real-time online information into anoff-line AR system.

The invention also describes a new way to structure and manage ARcontent. In this example, the AR content may be divided into one or morestages. Each stage of an AR source content item can be independentlymodified irrespective of the other stages of the AR source content orother item of AR content that are incorporated into an AR environment.One embodiment of the content management system described herein updatesone or more stages of the AR source content in real time byincorporating user AR content including real life images and live userinteractions with the AR source content. The content management systemprovides this user AR content to an AR machine so that the images andinteractions performed by the use can be incorporated into the ARenvironment in real time. In this way, the content management systemallows live user interactions with virtual 2D/3D holograms. In oneexample of this live interaction feature, a virtual 2D/3D hologram isgenerated in a item of AR source content. A user interacts with the2D/3D hologram by talking, moving, looking, or otherwise modifyingbehavior in relation or response to the 2D/3D hologram. The userinteracted creates user generated AR content. The content managementsystem stores and provides the user generated to an AR machine in realtime as the interaction is occurring. The live interaction is thensimultaneously incorporated into the AR environment by the AR machine toprovide a real time interactive experience between the user and the2D/3D hologram.

User generated content is unique for each user therefore the contentstructure needs to be flexible to allow for different user interactionswith the same content. The AR content of the invention described hereinis structured to have one or more stages. The stages are independent ofother stages in the same AR content item and other AR content items. Todeliver a live user interaction with a 2D/3D hologram, user generatedcontent containing the unique user interaction is augmented into thestage of AR source content containing the 2D/3D hologram at a particularorientation and position that received the user interaction. The userinteraction is added to this particular stage in real time to provide alive interactive experience between the user and the AR environment. Inone example, the content management system manages the AR source contentto restrict access to AR source content by users. Access restrictionspreserve the AR source content in its original state to allow otherusers to have their own unique interactions with the AR source content.The content management system stores AR user content independently fromthe AR source content so that users will always have access to theirsource content. Any time the user wants to replay the interaction withthe 2D/3D hologram the content management system will provide the ARuser data containing the interaction and the AR source data containingthe 2D/3D hologram and the AR environment to a AR machine for display.In one example, the content management system associates the AR userdata with a particular stage of AR source content so that the 2D/3Dhologram will be in the correct orientation and position to receive theuser interaction.

In another example, the content management system provides content for alive user interaction with the AR environment. In this example, the ARdevice detects the image of a user and updates the AR environment withthe user image. Movements by the user detected by the AR machine willappear in real time in the AR environment due to the operation of thecontent management system. Both user AR data including the live image ofthe user and AR source content data contain the virtual environmentcontain the user's picture are stored in a content database managed bythe content management system. To provide a live image of the user inthe virtual environment, the content management system provides thestage of AR source content corresponding to the AR user content, theuser image portion of the AR user content is also provided by thecontent management system. The AR machine then renders the stage of ARsource content and the AR user content image in real time to display alive image of the user in the AR environment.

In one example, the content management system receives raw data and ARsource content from an AR machine, checks the authorities dependenciesassociated with the data, for example, permissions, access restrictions,offline or offline availability that can include permissions and oroffline/online availability) and sends the data to other machines for ARsource content to combine with the data managed by the contentmanagement system.

Data provided by the content management system is sent to an AR machineso that users can use the AR source content. When users interact withthe AR source content they generate AR user content. The AR user contentmay be incorporated into other AR content and saved as user generated ARsource content or AR combined content that is saved in a contentdatabase managed by the content management system. The AR user contentmay also be stored in an uncombined form as AR user content managed bythe user. Accordingly, the content management system of the presentinvention may include an AR source content management system that is notaccessible externally by regular users and an AR user content managementsystem that is publicly accessible by regular AR users. The AR usercontent management system allows users to view and share their own ARuser content with other users so that the other users may incorporatedthe AR user content into their own AR environments.

AR combined content and user generated AR source content is managedthrough AR source content management system and the AR user contentmanagement system. User authorization for AR combined content and usergenerated AR source content is more permissive than AR source contentbecause regular users can access the AR combined content and usergenerated AR content they created.

The content management system for managing AR user content is furtherconfigured to manage AR user content for holographic AR. AR user contentin holographic AR systems always includes user images and interactionwith 2D/3D holograms displayed in the AR environment. The contentmanagement system comprised herein provides special memory for storingAR user content in short term storage so that it can be quicklyaccessible by AR machines displaying the AR user content in and ARenvironment. New AR source content interactions contained in the AR usercontent generate AR combined content and displayed as a live interactionbetween the user and the holographic 2D/3D image. The content managementsystem described herein further prevents the AR source content frombeing modified while also providing the AR source content augmented withAR user content to an AR machine in real time. This capability allowsusers to have different and unique interactions with the same AR sourcecontent.

Unique user interactions with AR source content generate a thirdcategory of AR content, AR combined content which provides auser-designer co-creation experience. Users interacting with AR sourcecontent in unique ways generate new AR source content along with theirown AR user content. The AR machine and content management systemdescribed herein combine to deliver a combined AR creation and ARexperience on the AR platform described herein. They dynamic,independently modifiable content structure allows for the combined ARcreation and AR experience by allowing unique user interactions with ARsource content to be upgraded into existing AR source content in realtime to deliver an AR environment with users can have live interactionswith virtual objects displayed in AR environments.

The AR user content generation further enjoys the benefit of flexibilityof updating one or more stages of AR source content independently fromother stages of source content to that the unique user interaction maybe incorporated into the stage it occurs in without terminating,restarting or otherwise interrupting the other stages of the AR sourcecontent. Accordingly, if an user or admin performs or restricts a uniqueAR interaction in the middle of a AR content stage, the existinggenerated AR user content is still saved and can be processed, replayedand shared without permanent or temporary interruption.

Data analysis results are managed in a different way. Data analysisresults that are releasable to users, for example, a user's own profileanalytics results and own connected device analytics results, have ARuser content authorization. Data analysis results that are notreleasable to users, for example business data analytics results oranonymized aggregated user profile analytics results or connected deviceanalytics results, are access restricted and require higher levelauthorization, for example, admin authorization, AR source contentauthorization or corporate client authorization

In this example, the AR source content is structured with stages. Eachstage can be independent from other stages. Therefore the stages can beupdated in real-time. Here stage is dynamic, flexible, andinterchangeable. Accordingly the AR source content is structured like aseries of movie scenes not rigid steps. The structure of the AR contentis uniquely flexible therefore the content management system describedherein requires additional complexity to efficiently management theunique content structure.

In one aspect, disclosed herein is a content management system forproviding real-time augmented reality (AR) data to one or more ARmachines. The AR machines receive a continous stream of raw dataincluding visual data, and optionally one or more of audio data, sensordata, or a some combination. The some portion of raw data is then sentto the content management system and data analtytics service. The dataanalytics service extracts user, device, or business insights from theraw data and geneates analytics results information that is sent to thecontent mangement system. The content management system users theanalytics results information to select and provide content to the ARmachine.

In this example, the analytics results information may contain at leastone of a user preference, a system setting, an integration parameter, acharacteristic of an object or a scene of the raw data, an interactiveuser control, or some combination. The content management system and thedata analytics service operate in real time to modify one or more stagesof AR source content based on the analytics results information, forexample, the content management system by provide less detailed ARsource content or strip away some objects contained the AR sourcecontent based on connected device analytics result information theindicates the connected device struggling the maintain the computationalperformance necessary to fully render the AR environment in real time.

In one example, analytics results information related to one stage of ARsource content is extracted in real-time from the raw data or a variantthereof at one or more time points based on analytics resultsinformation extracted on a previous or concurrent stage of the same itemof AR source content.

In another example, the analytics results information is learned basedon aggregated analytics results information extracted from multiple timepoints. In this example the data analytics service may deploy one ormore artificial intelligence systems or machine learning models to learnanalytics results information. This machine intelligence may be deployedas a machine learning or artificial intelligence model that can be usedto anticipate analytics results information, thereby improving theefficiency of the content management process and reducing the amount ofcomputational resources needed to generated a live interacted experiencewith an AR environment. The analytics results information describedherein may include many information types including text, one or morecodes, one or more numbers, one or more matrixes, one or more images,one or more audio signals, one or more sensor signals, one more machinereadable data types or some combination.

In some embodiments, the content management system combines raw data andat least one of one more stages of AR source content or AR user contentaccording to one or more data analytics results. The combinedinformation is then sent to an AR machine for display.

In some embodiments, the content management system removes unwanted datafrom the raw data, variant thereof, one or more stages of AR sourcecontent or AR user content, the unwanted data comprising unwantedenvironment scene data, data that are determined to be irrelevant basedon one or more criteria comprising a user preference, a system setting,a characteristic of an object or scene from the raw data, selected datafrom which information has been extracted, selected information fromwhich knowledge has been learned, or some combination.

In some embodiments, the at least one set of data analytics informationis of a different type from the data parameters in the raw data thatcorrespond to the information input or the knowledge input.

In some embodiments, the data analytics information is extracted fromthe raw data or AR content using big data techniques, for example datastreams, data frames, distribute storage, distributed processing anddatabase sharding.

In some embodiments, the data analytics service learns relevant databased on information extracted at multiple time points based on one ormore criteria, for example, a user preference, a system setting, acharacteristic of an object or scene from the raw data, an integrationparameter, an interactive user control, at least one criterion based onbig data, or some combination.

In some embodiments, the content management system provides compressionportions of real-time data to AR machines.

In some embodiments, the raw data received by the content managementsystem from a third party computer device or Internet application.

In some embodiments, the method or operation comprises: receiving, inreal-time at the computer device, existing AR-data from the othercomputer device, the existing AR-data comprising existing informationdata and existing knowledge data.

In some embodiments, the raw data received by the content managementsystem is generated by a data input and output component local to the ARmachine generating the raw data. The local data input and outputcomponent may include a microphone, a camera, a display, a sensor, orsome combination.

In some embodiments, the camera comprises a 2D camera, a 2D/3D camera, a4D camera, a color camera, a near infrared camera, an infrared camera, athermal camera, a multi-spectra camera, a hyperspectral camera, or somecombination.

It would be understood that any embodiments disclosed herein can beapplied, when applicable, in any aspect of the invention, alone or inany combination. Details of one or more implementations are set forth inthe accompanying drawings and the description below. Other features,aspects, and potential advantages will be apparent from the descriptionand drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a client-server environment of an AR platform forgenerating live interactive AR environments.

FIG. 2 illustrates a server system arrangement for managing AR contentusing real time data analytics.

FIG. 3 displays an example process on a computer for managing contentitems using data analytics information and user access control settings.

FIG. 4 illustrates an example process on a computer for performingdifferent types of data analytics on raw data and AR content.

FIG. 5 displays an example process on a computer for managing one ormore content elements using a content management system.

DETAILED DESCRIPTION

As used herein, the term “AR source content” means any AR content itemcomprising audio, video, sensor data, interactive control information,or some combination. As used herein, the term “AR user content” meansany AR source content user interaction data describing how one or moreusers interacts with AR source content. AR user content may includemetadata or any other publicly or privately accessible user interactioncontent or other user generated content. As used herein, the term “ARcombined content” includes user generated content that includes a novelinteraction with AR source content. AR combined content may includeelements of AR user content and AR source content that includes and maybe packaged as content metadata or another form of AR content.

FIG. 1 illustrates a client server arrangement for providing aholographic AR environment. The client server arrangement may alsoprovide for live user interaction with an AR environment, for example,what a 2D/3D hologram rendered in the AR environment. The arrangementincludes one or more client side components 100 that interact with oneor more server side components 120 using an application interface 110,for example an application programing interface (API) written in aprograming language, for example, PIIP, Python, Java, Node, orJavaScript.

The client side components 100 are implemented in a web based or mobileapplication programmed to run on a plurality of computing devices forexample, a desktop computer, laptop, tablet, mobile phone, or smartphone. The client side components include a communications module 101that provides a wireless service connection 102 for interfacing with theserver side components 120, one or more internal components, third partyservices, or computer systems, for example, 130-137, or otherapplications connected to the Internet. Information received from thewireless 102 service connection is provided to the graphical userinterface (GUI) 105 for display and further processing. The imagingengine 104 generates 2D/3D models, simulations, and AR environments thatprovide interactive AR experiences and live user interactions with 2D/3Dholograms. The imaging engine 104 interfaces with one or more renderinglibraries 103 to compile, arrange, and/or buffer one or more objectsgenerated by the imaging engine 104 into a static or dynamic ARenvironment that provides for live user interaction. In one example, thedynamic AR environment is an intergalactic travel experience thatdepicts users traveling to different planets in the solar systemincluding Mars, Earth, Jupiter, and Venus. The intergalactic travelexperience may also include scenes from other elements of the universe,for example stars, black holes, and moons. Another example includes atransitional AR environment that transitions between a stage of anintergalactic travel scene and an advertising aspect from a sponsor. Oneor more objects provided by the imagine engine 104 may also be processedby one or more rendering libraries 103 to generate an augmented realityenvironment. The rendering libraries 103 may interface with the GUI 105to present an augmented reality environment as an interactive model of aBlue Whale hologram and a live user image transposed in an underwater ARenvironment that supports live user interactions with the Blue Whalehologram. In one example, the user interacts with the GUI 105 to angle,rotate, or otherwise manipulated the hologram by moving a hand or otherbody part in the direction of the hologram. The augmented realityenvironment provided by the GUI 105 tracks changes in body position andautomatically adjusts the user image and the 2D/3D hologram to reflectthe changes. Accordingly, the augment reality environment provides arealistic live user interaction with 2D/3D holograms rendered in an ARenvironment across a full range of rotational motion, spatialarrangements, position alignments. Multiple cameras may be used torender 2D/3D holograms of virtual objects and or extract images of realpeople into virtual reality environments to provide live userinteractions with AR environments.

The components 120 included in the server system may be configured torun on one or more servers, virtual machines, or cloud computeinstances. The server side components 120 include a communicationsmodule 124 that provides a connection to a wireless service 127 as wellas a network connection and/or security layer 128 connection forauthenticating access to a network for interfacing with the client sidecomponents 100, the application interface 110, one or more internalcomponents, third party services or computer systems, for example,130-137, or other applications connected to the internet. The serversystem further includes a content management system 122 managingdocuments, graphic content, artificial intelligence models, 2D/3Dmodels, simulations, and augmented reality environments and othercontent produced or processed by the server side components 120. Datastorage 121 provides physical storage, memory, and backups for datamanaged by the content management system 122 and generated or processedby one or more server side components 120. One or more imaging engines125 generate one or more 2D/3D models, simulations, or augmented realityenvironments from data provided by the content management system 122. Inthis example, the content management system may provide many differenttypes of data, for example, user generated content, user AR interactivecontent, pre-designed AR content, real-time content, overlayinformation, voice and music information, sponsor information, forexample, advertising aspects or offers, advertisement biddinginformation, social media information, data privacy configurations, anddata access control settings.

The data analytics service 126 may provide analytics information thatdetermines the content provided by the content management system 122.Analytics information provided by the data analytics service 126 mayalso influence the stages of AR content or content elements integratedinto AR source content by the imaging engines 125. After receivingcontent from the content management system 122, the imaging engines 125may add, remove, or update one or more content elements within a stageof AR source content or multiple AR source content stages based onanalytics results data and other instructions received from theanalytics service 126. Once the content elements or stages of AR sourcecontent are updated, rendering libraries within the imaging engines 125interfaces with the imaging logic contained in the imaging engines 125to provide one or more 2D/3D models, simulations, or augmented realityenvironments to the application interface 110 for processing and displayon one or more client side applications running a plurality of clientdevices.

The server system further includes business logic 123 for performing theday-to-day business tasks of the server and client systems. Tasksperformed by the business logic include data analytics, accounting andpayment processing, as well as chat and messaging.

One or more third party services, computer systems, or otherapplications connected to the Internet may provide data to- or otherwiseinterface with- at least one of the client side components 100 or theserver side components 120. Example internal system components and thirdparty services include sensors 133-137 and component input and outputdevices 130-132. Example sensors include a temperature sensor 133 fordetermining user body temperature or the air temperature of the reallife environment surrounding a user and a motion sensor 134 for trackingphysical movements of the user including changes in body position,facial expression, as well as gaze direction and duration. The motionsensor 134 may track movements of an individual user and well as themovements of several people if the user is in a busy area or part of acrowd. Other sensors provided in this example include a IR sensor 135for sensing infrared wavelength light to measure connected deviceperformance and interaction data as well as perform image analysis, apressure sensor 135 for measuring air pressure and sound intensity, anda light sensor 137 for measuring light intensity to perform imageanalysis and determine the current time of day of a user. The dataanalytics service 126 can make automated inferences from processing rawsensor data, for example time of day, current environment weather, userinformation, connected device data, aggregate crowd information,business information, user environment information, or some combination.

Input and output devices that included in the system described hereinmay include a camera 130 for capturing live user images and video, amicrophone 131 for capturing audio input data such as user speech ormusic as well as speech and music from the environment surrounding auser, and a speaker 132 for providing audio data, for example, speech ormusic to a user or an AR environment. The input and output devicescapture user interaction information necessary to provide live userinteractions with AR environments. Information received from the inputand output devices can be processed by the data analytics service 126,stored and selectively provided by the content management system 122 andincorporated into one or more elements of a stage of AR source contentor several AR stages by the imaging engines 125.

Input and output devices and sensors interface with the server sidecomponents with through wireless and wired communications, for example,Wifi, Bluetooth, cellular communication including LTE, 5G, 4G, 3G, etc.,a local access network (LAN) or wide area network (WAN). A wirelessserver local to the server side components may also enable one or moreinput and output devices or sensors to interface with Internet basedapplications or realize some web-enabled functions, for example, remoteaccess and control, communication with one or more online messagingservices, or cloud based data storage and processing.

FIG. 2 illustrates a server system for generating 2D/3D models,simulations, and augmented reality environments that provide live userinteractions with hologram objects and AR environments. The serversystem includes a plurality of server side components including acontent management system 200, a data analytics service 220, anapplication interface 230, and an AR machine. A communicationsnetwork/security module 241 provides hosts a wireless server 242 toprovide a wireless connection for interfacing with cloud based virtualmachine instances and applications connected to the Internet. Thecommunications network/security module also provides a networkconnection for interfacing with third party services, computer systems,or other applications through wired connections. The communicationsnetwork/security module 241 also provides security features includingnetwork security and platform authentication to prevent regular usersfrom accessing AR source content, local machine data 246, businessanalytics information and other limited access platform data stored inthe data storage module 243 on an AR machine 240 or the analytics resultdatabase 226 inside the data analytics service 220. Platformauthentication services provided by the communications network/securitymodule 241 also provide users access to user generated content 245created by the user and other user information such as user profileinformation, user preferences, user image data, and user interactionswith digital objects and AR environments.

The communications network/security module 210 interfaces with theimaging engines 260, data analytics service 220, and content managementsystem 220 to provide secure data received from one or more Internetapplications/cloud computing instances 270 that interact with the seversystem, for example, social media applications, Internet searchapplications, third party APIs with a web endpoint, cloud storageinstances, or cloud compute virtual machine instances. The contentmanagement system 200 interfaces with the Internet applications/cloudcomputing instances 270 via a wireless service connection 206. Using thewireless service connection 206, the content management system mayincorporate real time online content and other content from Internetapplications or cloud computing virtual machine instances into the ARsource content database 205. The wireless service connection 206interfaces with the access control module 201 to restrict public accessto the AR source content database 205 and grant public user access to ARuser content items created by the use that are stored in the AR usercontent database 204.

The content management system 200 also includes content management logic203 for selecting one or more content elements, content items, or stagesof a content item from the AR source content database 205. The contentmanagement logic 203 may also select one or more elements of AR usercontent or AR user content items, for example, a user image or userinteractive data, from the AR user content database 204. Selected ARsource content and AR source content items are sent to the imaginingengines 260 inside an AR machine 240. The imaging logic 267 then updatesthe AR source content with the content provided by the contentmanagement system 200 and the rendering libraries 268 read the AR sourcecontent and render AR environment and virtual objects as instructed bythe AR source content.

In one example, the access control module interfaces with the AR sourcecontent database 205 to prevent regular users from accessing the ARsource content items stored in the content database 205. By restrictingaccess to the AR source content 205 the access control module 201ensures the many users will be able to interact with the same AR sourcecontent without altering or corrupting the experience of other users.The access control module 201 allows admins and other users withhigher-level AR source content database permissions to add, remove, ormodify AR source content items stored in the AR source content database205.

The content management system 200 interfaces with the data analyticsservice 220 through an application programming interface (API) 210. Thedata analytics service 220 ingests raw data from one or more sensors 250and input output devices, for example, a camera, microphone, or speakerand AR content from the data storage module 243. One or more analyticsmodules 221-223 and 225 then process the data to generate analyticsresults information. In one example, the user analytics module 221generates user analytics information, for example, user identificationinformation, user engagement data, or user interaction information fromprocessing one or more of user AR content, image data from a camera, andmotion sensor data. In another example, the connected device analyticsmodule 222 generates connected device analytics information, for exampleconnected device identification, connected device interactioninformation, and connected device performance information, fromprocessing one or more of device interaction information, AR sourcecontent display information, or AR user content. The brand analyticsmodule 223 generates business analytics information, for example useradvertising aspect engagement, aggregate crowd advertising aspectengagement, or advertising aspect conversion rate from processing userand aggregate crowd motion detection information, and connected deviceinformation such as user browser history or user purchase history.

The Data analytics service may also contain a artificial intelligencemodule 225 for learning data analytics information from user, userenvironment, connected device and brand training data collected overtime. By learning analytics information and distilling these learnedinsights into one or more artificial intelligence models or machinelearning algorithms the artificial intelligence module 225 may predictanalytics results for a new user or connected device with fewerprocessing steps and computational resources than other means of dataanalysis.

Analytics results information generated by the data analytics service220 is stored in an analytics results database 225. The authenticationlogic 224 restricts access to the analytics results database 226 toprevent regular users from obtaining analytics results on other users orconnected devices. Accordingly, the authentication logic 224 willrestrict access to the analytics results database to admins or userswith higher-level authorization, for example, business accounts orcorporate clients conducting brand analytics on an advertisementcampaign run on the AR platform.

Analytics results information generated by the data analytics service220 is provided to the content management system via and API 210. Theanalytics results information indicates content that may be selected bythe content management system 200 to provide to the AR machine 240. Inone example, analytics results information indicates a user has a highengagement with advertising aspects that are red colored and containcute dogs. Further analytics results information indicates the AR sourcecontent currently being rendered on the user's display device includesan advertising aspect that has a fast car overlaid on a blue background.To modify the advertisement aspect of the AR source content the ARcontent management system 200 selects AR source content having cute dogsand red color from the AR source content database 205. The contentmanagement logic 203 may also identify the stage of the AR sourcecontent and the elements of the stage advertising aspect that need to bemodified. The AR content management system then sends the AR sourcecontent to the AR machine 240 along with the modification instructions.The imaging engines 260 then incorporate the AR source content form thecontent management system into the AR source content displayed on theuser device in real time so that when the user reaches the stage of theAR source content containing the advertising aspect the imaging engines260 will render the modified version of the AR source content containingthe advertising aspect the user prefers.

An AR machine 240 interfaces with the content management system 200 anddata analytics service 220. The AR machine includes one or more sensors250 and input output devices for extracting live data from users andtheir surrounding environment. In one example the sensors 250 include alight sensor 251 for measuring light intensity and colors, and IR sensor252 for measuring infrared wavelength light emitted by one or more userdevices in the vicinity of the AR machine, and a motion sensor 253 formeasuring changes in physical position and facial expression buy a useror crowd of users. One or more additional sensors 254, for example, apressure sensor, a temperature sensor, or a biometrics sensor, may alsobe incorporated into the AR machine 240 to collect additional raw dataabout users and their surrounding environment. Raw data collected by thesensors 250 is sent from the AR machine 240 to the Data AnalyticsService 220 for further processing to extract insights from the data.Additional platform data, for example, AR content data including thenumber, type and history of AR content played as well as performancedata related to how a connected device performed with rendering and ARenvironment may also be provided to the data analytics service 220 bythe AR machine 240. User interaction data, for example the number, type,and frequency of user interactions with particular AR environments anddigital objects within AR environments may also provided to the dataanalytics service 220 by the AR machine 240.

The AR machine 240 further contains imaging engines for modifying,creating, and rendering AR content as a live interactive AR environment.The imaging engines include rendering libraries 268 that interfaces withthe imaging logic 267 to provide one or more 2D/3D objects, models,holograms, simulations, animations, or augmented reality environments toan application interface. In one example, the rendering libraries 268includes 2D/3D object rendering libraries for compiling 2D/3D objectsgenerated by the 2D/3D modeling engine and AR rendering libraries forcompiling AR environments generated by the AR engine. The renderinglibraries 268 further include simulation streaming libraries forstreaming 2D/3D object animations and user interactions provided by theanimation engine over a content streaming network as well as imagerecognition logic 261 for matching the orientation of a live user imagein an AR environment with the orientation of the user in real life. Inone example the image recognition logic 261 recognizes a user from antheir image and user movements in real life using recognition logic thatis specific to the user and learned over time so that it becomes moreprecise the more times the user operates the platform software.

The imaging engines, 260, further include image tracking logic 262having one or more libraries for tracking movement of the user in livestreaming video and automatically adjusting the image of the userdisplayed in the AR environment to match changes in user physicalposition and facial expressions that occur in real life. AR contentincluding 2D/3D objects, models, holograms, animations, simulations, andAR environments generated by the imaging engines 260 are managed by thecontent management system 200.

The imaging engines 260 also include 2D/3D object generation logic 263for creating holograms that are displayed in holographic ARenvironments. The 2D/3D object generation logic 263 includes hologramimagining libraries that generate the physical appearance of thehologram object as well as hologram physics libraries for generatinghologram movements that are similar to movements of real world objectshaving a similar size, shape, and physical mechanics. The 2D/3D objectgeneration logic interfaces with the interaction recognition logic 264to provide live interactions between users and holograms rendered in aAR environment. The interaction recognition logic 264 includespre-designed libraries of user physical movements and facial expressionsthat produce responses from the 2D/3D hologram object rendered in the ARenvironment. In one example, having 2D/3D hologram spaceship theinteraction recognition logic 264 is instructed to place the user imageinside the spaceship with the user jumps in the air and turn thespaceship to the left if the user is already in the spaceship and raisestheir left arm.

In another example the interaction recognition logic 264 allows the userto generate their own unique interactions with 2D/3D hologram objects inreal time. In this example, the interaction recognition logic 264detects a new undersigned physical movement performed by a user. Theinteraction recognition logic 264 then communicates the new userinteraction to the 2D/3D object generation logic 263. The 2D/3D objectgeneration logic 263 then generates an appropriate response by the 2D/3Dhologram object. The user interaction is saved as new AR user contentand the 2D/3D hologram response is saved as new AR source content by thecontent management system 200. The content management system 200 sendsthe AR source content and AR user content to the content immersionlibraries 265 that contain instructions for updating the AR sourcecontent currently rendered by the AR machine 240 with new AR content.The environment augmentation libraries then augment the new userinteraction and 2D/3D hologram response into the AR environmentcurrently displayed on the AR machine 240.

In this example the AR source content is divided into stages so elementsof one stage can be updated quickly without affecting the other stages.This content structure allows the AR user content containing the newinteraction and the AR source content containing the new 2D/3D hologramresponse to be incorporated by the content immersion libraries 265 intothe AR source content currently being rendered on the AR machine in realtime. Similarly, the flexible AR source content structure also allowsthe environment augmentation libraries to update the environmentrendered on the AR machine in real time with the new user interactionand new 2D/3D hologram response. The real time AR content augmentationand new AR content immersion allows the content management system 200described herein to provide a live user experience where the user caninteract with one or more 2D/3D hologram objects in unique ways in realtime within an AR environment.

FIG. 3 displays an example workflow for using an augmented realitysystem to provide a live interactive experience. In this example, An ARdevice 300 measures raw data 301 from sensors, for example, a motionsensor or light sensor and input and output devices, for example acamera or microphone. The AR device 300 further aggregates AR contentmetadata 302, for example number, of views or device performanceinformation and AR interactive data 303, for example, user interactiondata or user generated AR content. The raw data 301, AR content metadata302, and AR interactive data 303 are sent from the AR device 300 to thedata analytics service 305 for processing. The data analytics service305 may also incorporate information from a third party service orInternet through a wireless connection with an analytics cloud server ofinternet application 304.

The data analytics service 305 processes at least one of raw data 301,AR content metadata 302, and AR interactive data 303 to generateanalytics information. In this example, analytics information mayinclude user analytics 307 results such as user identificationinformation and user engagement or response metrics. Connected deviceanalytics results 308 may include, for example, device performancemetrics or device interaction metadata. Business analytics 309 mayinclude advertisement aspect effectiveness metrics or aggregate userpurchase history for a particular product or subset of products.Environment analytics results 310 may include aggregate user informationsuch as crowd response metrics or pedestrian/person count measurements.Analytics information generated by the data analytics service 305 may bestored in an analytics database 306. An access control module mayrestrict access to the analytics database 306 to prevent regular usersfrom accessing business analytics. Optionally, the data access controlmodule may also restrict access to the analytics database to preventcorporate users from obtaining personal information, such as name,address, telephone number, IP address, device MAC address, or personalidentification number, for example, social security number or stateid/drivers license number. The analytics information may also be storedin memory, refactored as program instructions or otherwise set to thecontent management system 311.

In this example, the content management system 311 recieves theanalytics information and uses it to select one more stages or elementsof at least one content item stored in a content database 313. Thecontent management system 311 may also provide content from a thirdparty service or Internet application through a wireless connection witha content cloud server or Internet application 312. Additionalinformation incorporated from the content cloud server of Internetapplication 312 may include social network information or content mediaas well as content media and information obtainable through an Internetbased web search.

Content retrieved from the content database 313 or Internet by thecontent management system 311 is provided to the AR device 300, the ARdevice 2 315, a plurality of other AR devices, or some combination.Content provided by the content management system 311 is thenincorporated into one or more stages of AR source content to provide alive user interactions with an AR environment, for example, live userinteractions with a book, teacher hologram, or other students in aclassroom virtual environment. A data access control module 314 mayrestrict access to the content database so that regular users cannotaccess AR source content and users can access AR user content, ARcombined content, or AR user content created by them.

FIG. 4 illustrates an example process on a computer for performingdifferent types of data analytics on raw data and AR content. In thisexample, the data analytics service 400 receives raw data and AR content401. AR source content may comprise metadata and user interactioninformation for at least one of type of AR source content, AR usercontent, AR combined content, user generated AR content or somecombination. The analytics service 400 then performs data analytics togenerate analytics results information 402. In this example analyticsresults information includes user analytics results 410, connecteddevice analytics results 420, business analytics results 430, andenvironment analytics results 440.

The user analytics results 410 may include user identification 411, userbiometrics 412, user behavior information 413, user AR interactiveinformation 414, user expression information 415, user gaze trackinginformation 416, user social media information, or some combination.Connected device analytics results 420 may include connected deviceidentification 421, connected device user association 422, connecteddevice AR interaction 423, connected device performance metrics 424,connected device location tracking 425, or connected device browserhistory 426. The business analytics results 430 may include user recentpurchase history 431, user advertisement engagement 432, advertisementperformance 433, AR advertisement bidding 434, AR advertisement returnon investment 435, AR advertisement budgeting 436, user or locationSocioeconomic information 437, or some combination. The environmentanalytics results may include data and metrics relating to aggregatecrowds interacting with an AR environment, for example, pedestriancounting 441, crowd engagement 442, aggregate user information 443,aggregate device information 444, aggregate social media information445, proximity information 446, or some combination.

Other example embodiments of the data analytics service are describedherein. In one example, the content management system includes a dataanalytics service. The analytics service may process data provided tothe content management system including data stored in the contentdatabase. The analytics service may also process information receivedfrom a third party Internet application, for example, a social medianetwork, an internet search application, or a third party data providerhaving a data source that is accessible over an application programinginterface (API) having an endpoint connected to the Internet, forexample a virtual machine instance endpoint having a wireless or networkInternet connection or an virtual machine instance connected to awireless Internet server. The data analytics module may also processinformation from any internal datastore, for example, a contentdatabase, a local database on an AR machine, or a platform database on aserver side system.

One function of the analytics module is to determine user and deviceinformation associated with content items handled by the contentmanagement system. Analytics results information generated by theanalytics module are then used by the content management system todetermine content items to provide back to an AR machine. In oneexample, the data analytics module processes one or more items of usergenerated content to identify the user who generated the content. Inanother example, the data analytics service processes AR user content todetermine users response to an AR environment or an object displayed inan AR environment, for example, a 2D/3D holographic object. Theanalytics service may extract other user preferences or responseinformation from sensor data collected by an AR device. In one examplethe analytics service determines user preferences for analyzing at leastone of user expression information, user biometrics information, user ARinteraction information, user gaze tracking, user dress and wearablesinformation.

In one example, the data analytics service processes sensor informationto determine user response to an advertising aspect of an ARenvironment. To determine user engagement with the advertising aspectthe data analytics module extracts user identification informationincluding demographics information, for example, gender, age, andethnicity, form processing video or images received from the AR machine,Facial recognition, retina scans, or other biometrics information mayalso identify the user. As part of this process the biometricinformation may be automatically associated with a particular userprofile or if no profile is associated with the biometric information anew user profile may be automatically created and associated with thebiometric information. In addition to extracting user identificationinformation, the data analytics module may further processes receivedvideo or images to determine use engagement with- and attention to- theadvertising aspect by tracking user gaze direction and duration. Thedata analytics module further determines user sentiment towards theadvertising aspect by analyzing user facial expressions to determine ifthe user reaction to the advertising aspect was positive, negative, orindifferent.

Analytics results from this example analysis may stored in the analyticsmodule and protected by an access control module. The access controlmodule restricts access to the analytics results information to preventpublic access. In this example, access to the results informationcontaining the user engagement, sentiment, and response to theadvertising aspect is provided only to a business or corporation runningthe advertising aspect within the AR environment. The user identifyportion of the analytics results information may be hashed or otherwiseobfuscated to keep confidential the identity of the user pictured in thevideo or image processed by the data analytics service. After processingto generate analytics results information the video or images containingthe user may also be deleted to further protect user privacy. Similardata processing techniques may be used to identify users and determineuser engagement, sentiment, or response to A variety of AR environments,for example, an educational environment, a practical skills trainingenvironment, and business information environment, an entertainmentenvironment, or a medical environment.

Analytics results information may also determine how to modify contentitems provided back to an AR machine. Modifications may includeadditions, changes, deletions, or some combination. Modifications alsoinclude new combinations of one or more stages of AR source content. ARsource content, AR user content, AR combined content, and AR usergenerated content may also be incorporated into one or more stages of ARsource content according to analytics results information generated bythe analytics module. Analytics results information may be generated forany stage of AR source content and analytics results generated for onestage are independent of analytics results form other stages.Accordingly, one or more stages of the same AR source content item maygenerate different analytics results information when processed by theanalytics module.

In addition to user data the data analytics service also processesextracts device information from devices connected to an AR machine orother AR system component. In one example, extracted device informationincludes connected device identification, connected device interactioninformation, and connected device performance information. By extractingseveral categories of information on devices connected to AR machines,the data analysis service provides insights into how performance of theAR machines can be optimized on one or more connected devices. Theconnected device analytics results information may also be used toprovide additional information on user engagement and response to ARsystems.

In one example, the data analytics system extracts connected deviceidentification information and updates a device profile or if noneexists creates a new device profile. The data analytics may then extractinformation from the connected device to gain more insights into userengagement and response to a particular AR environment, for example, thenumber of times the user replayed a piece of AR content on the connecteddevice or the amount of time the user spent in each AR environmentdisplayed on the connected device. Other information including thenumber of AR user content shared by the user or the websites the userbrowses after viewing or interacted with AR source content providefurther information about the effect of the AR environment on the user.In one example, connected device information may be used to helpadvertisers track the effectiveness of an advertisement aspect embeddedin an AR environment. In this example, the number of users who browsedfor goods mentioned in the advertising aspect or related to theadvertising aspect can be tracked using device information with highconversion rates between mentioning a particular product in anadvertising aspect embedded in an AR environment and browser hits onsits selling the product mentioned is good evidence of a successfuladvertising campaign that should be continued or expanded.Alternatively, low conversion rates provide evidence that theadvertising aspect was ineffective and should be discontinued ormodified.

In another example, the data analytics service may use connected deviceperformance analytics results to modify AR content provided to an ARmachine. Connected device analytics results indicating the connecteddevice lacks the computational performance to smoothly render the ARenvironment may cause the content management system to provide anincomplete or less vibrant version of the AR source content to the ARmachine. Alternatively, connected device performance analytics resultsindicating the connected device is easily rendering an AR environmentcould cause the content management system to deliver a more complex orheavily augmented version of the AR source content to the AR device.

The flexible structure of the AR source content further increases thecustomizability of the AR environment provided by the content managementsystem. In one example, the data analytics service generates useranalytics results indicating a user has a high preference for mountainlandscapes. The analytics service further indicates the connected deviceassociated with the user has a very high computational performancecapacity. The content management receives this information and providesAR source content to the user that contains several stages of theentertainment AR environment selected by the user with an stagecontaining a brief cut away to advertisement aspect stage containing themost vibrant mountain landscape in the content database and anadvertisement for bottled water.

In this example, the data analytics service may also incorporatebusiness intelligence information such as the highest bidder for aparticular demographic of customer using the AR platform or thecategories of product a particular user has browsed on the associatedconnected device in the last month. This business information can beupdated in real time and incorporated into AR environments in real timeby the content management system to allow maximum flexibility,personalization, and brand message enhancement. The data analyticsservice also ingests raw data from one or more sensors in an AR machineto extract user environment information, for example, pedestriancounting, people counting, geographic location, socio-economic status ofa particular location, and local information including business andlandmarks near the current location of the user. This additionalinformation provides more ways to customize the AR content provided bythe AR content management system.

FIG. 5 describes a process for updating one or more stage or elements ofa stage of AR content using the content management system describedherein. In this example, the content management system 500 receivesanalytics results information form the data analytics service 501. Thecontent management system 500 then selects content or an element ofcontent from the content database 502 using the analytics resultsinformation. The content elements stored in the content databaseinclude, for example, user generated content 504, user AR interactivecontent 505, pre-designed AR content 506, real-time content 507, overlayinformation 508, voice and music information 509, sponsor information510, advertisement bidding information 511, social media information512, user privacy configurations 513, data access control information514, or some combination. One or more of these content elements areselected from the content database and provided to an AR machine 515.The AR machine augments one or more stages of AR source content byupdating the stages of AR source content with the received contentelements 516. The updated AR source content is then rendered in an ARenvironment 517. The AR environment may be provided to users in realtime to give a live interaction user experience, for example, a liveinteraction with a teacher, instructional material, fellow student, orother digital object generated in a classroom augmented realityenvironment.

We claim:
 1. A method for managing content comprising: receivingaugmented reality (AR) source content in a first content database, theAR source content having a plurality of stages, wherein a first stage inthe AR source content comprises: user image processing info in thisstage, comprising an integration instruction to incorporate the user(s)image into an AR user content in real-time in this stage, wherein theuser(s) image is extracted in real-life in real-time, and at least onetransition criterion from this stage to another stage based on a user'sbehavior detected from real-time user image; and a second stage in theAR source content comprises user advertisement engagement info; storingat least concurrent stage of AR source content; analyzing at least onecontent item in the AR source content to extract content selectioncriteria; and providing at least one content item from a second contentdatabase, the provided content item being selected using the contentselection criteria, wherein the provided content item is incorporatedinto the stages of AR content in real-time.
 2. The method of claim 1,wherein the content items stored in the first content database compriseAR source content; and the content item in the second content databasecomprises non-AR raw data, AR source content, AR user content, or ARcombined content, the AR combined content comprising AR source contentand AR user content.
 3. The method of claim 1, wherein a stage in the ARsource content further comprises; virtual environment processing info;virtual object(s) processing info; interactive control informationincluding AR process controls programmed to occur in response tophysical actions of one or more users, wherein the AR process controlsare pre-defined, real-time learned, or some combination; user info,including user identification, user biometrics, user behavioralinformation, user AR interactive information, user expressioninformation, user gaze tracking information, user social mediainformation, or some combination; user authentication info; connecteddevice info, including connected device identification, connected deviceuser association, connected device AR interaction, connected deviceinformation, connected device location tracking, connected devicebrowser history, or some combination; business info, including userrecent purchase history, user advertisement engagement, advertisementperformance, AR advertisement bidding, AR advertisement ROI, ARadvertisement budgeting, socioeconomic information, or some combination;real-life environment info, including pedestrian counting, crowdengagement, aggregated user information, aggregated device information,aggregated social media information, proximity information, or somecombination; or some combination.
 4. The method of claim 2, whereindifferent AR user content is generated for different user interactionswith the same AR source content.
 5. The method of claim 1, wherein thestages of the AR source content are independently modifiable so whilerunning the current stage, all other stages can be updated withoutaffecting the operation of the current stage of the AR source content.6. The method of claim 1, wherein the content items incorporated intothe stages of AR source content comprises AR combined content comprisingone or more stages of user generated AR source content created fromunique user interactions with the AR source content.
 7. The method ofclaim 6, further comprising: receiving user generated AR source contentand updating one or more stages of AR source content with the new usergenerated AR source content.
 8. The method of claim 7, wherein the usergenerated AR source content is generated and incorporated into one ormore stages of AR source content to provide live user interaction withan AR environment.
 9. The method of claim 1, wherein the contentselection criteria comprises user preferences, user information, useridentification, user behavior, user bio-information, user ARinteraction, user facial expressions, user physical movements, user gazedirection, user gaze duration, user connected device information, userconnected device interaction, user dress, user social media data, userrecent purchase history, user social-economic status analytics,pedestrian counting, aggregated data analytics, people counting,location information analytics, or other analytics information.
 10. Themethod of claim 1, further comprising restricting access to at least oneitem of the AR source content or the content database(s), according toat least one authentication criterion.
 11. The method of claim 10,wherein the authentication criteria comprises: admin access information,user access information, user sharing preferences, user privacypreferences, organization access information, organization sharingpreferences, organization privacy preferences, online accessinformation, or offline access information.
 12. A content managementsystem comprising; one or more processors; and a memory accessible tothe one or more processors, the memory storing instructions executableby the one or more processors to: receive augmented reality (AR) sourcecontent in a first content database, the AR source content having aplurality of stages, wherein a first stage in the AR source contentcomprises: user image processing info in this stage, comprising anintegration instruction to incorporate the user(s) image into an AR usercontent in real-time in this stage, wherein the user(s) image isextracted in real-life in real-time, and at least one transitioncriterion from this stage to another stage based on a user's behaviordetected from real-time user image; and a second stage in the AR sourcecontent comprises user advertisement engagement info; store at leastconcurrent stage of AR source content; analyze at least one content itemin the concurrent stage of AR source content to extract contentselection criteria; and provide at least one content item from a secondcontent database, the provided content item selected using the contentselection criteria, wherein the provided content items are incorporatedinto the current stage of AR content in real-time.
 13. The contentmanagement system of claim 12, wherein the stages of the AR sourcecontent are independently modifiable so that while running the currentstage, all other stages can be updated without affecting the operationof the current stage of the AR source content.
 14. The contentmanagement system of claim 12, wherein the content items in the firstcontent database comprises AR source content; and the content items inthe second content database comprises non-AR raw data, AR sourcecontent, AR user content, or AR combined content, the AR combinedcontent comprising AR source content and AR user content.
 15. Thecontent management system of claim 12, further comprising restrictingaccess to at least one item of the AR source content or the contentdatabase(s), according to at least one authentication criterion.
 16. Anapparatus for content management comprising: memory for storing acontent management application and at least one stage of AR sourcecontent, the content management application programmed to manage ARcontent, wherein a first stage in the AR source content comprises: userimage processing info in this stage, comprising an integrationinstruction to incorporate the user(s) image into an AR user content inreal-time in this stage, wherein the user(s) image is extracted inreal-life in real-time, and at least one transition criterion from thisstage to another stage based on a user's behavior detected fromreal-time user image; and a second stage in the AR source contentcomprises user advertisement engagement info; processors for: receivingaugmented reality (AR) source content in a first content database, theAR source content having a plurality of stages; analyzing at least onecontent item in the concurrent stage of AR source content to extractcontent selection criteria; and provide at least one content item from asecond content database, the provided content item selected using thecontent selection criteria, wherein the provided content items areincorporated into the current stage of AR content in real-time.
 17. Theapparatus of claim 16, wherein the stages of the AR source content areindependently modifiable so that while running the current stage, allother stages can be updated without affecting the operation of thecurrent stage of the AR source content.
 18. The apparatus of claim 16,wherein the content selection criteria comprises user preferences, userinformation, user identification, user behavior, user bio-information,user AR interaction, user facial expressions, user physical movements,user gaze direction, user gaze duration, user connected deviceinformation, user connected device interaction, user dress, user socialmedia data, user recent purchase history, user social-economic statusanalytics, pedestrian counting, aggregated data analytics, peoplecounting, location information analytics, or other analyticsinformation.
 19. The apparatus of claim 16, wherein different AR usercontent is generated for different user interactions with the same ARsource content.
 20. The apparatus of claim 16, wherein the content itemsstored in the first content database comprise AR source content; and thecontent item in the second content database comprises non-AR raw data,AR source content, AR user content, or AR combined content, the ARcombined content comprising AR source content and AR user content.